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PREF ACE 


This thesis is concerned with the application of statistical methods 
to naval operational testing, and more specifically, with testing of the 
type conducted by OpDevFor and similar testing agencies. * The statis- 
tical methods considered are those of confidence limits, sequential 
analysis and the more recently developed Statistical Decision Theory, 
These techniques are regarded as of intergst to Naval line officers in 
various categories of billets, particularly Project Officers at testing 
agencies and officers concerned with planning which is based on the 
results of testing programs, Peegeentlee such officers are unfamiliar 
with the use and limitations of the three methods and the econ of 
these methods to each other, 

Statistical Decision Theory, in particular, can be useful at higher 
levels of the naval establishment suchas the offices of CNO. Naval 
planners at this level may be faced witha difficult problem in connec- 
tion with testing programs whose objective is the estimation of the per- 
centage effectiveness or probability of success of a weapon, in future 
combat, Such programs are costly to conduct, and increasingly so 
when an expensive weapon is tested to destruction. Costs ofa different 
nature are those associated ail the possible consequences if a poor 
estimate of the weapon's effectiveness is obtained from tests, The fun- 
damental problem is to determine how many trials are to be conducted 
and hence how many weapons snould be tested, Attempts to solve the 


problem by reconciling the conflicting costs will generally lead toa 
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dilemma, The application of Statistical Decision Theory to this problem 
is contingent upon the ability of the planner to specify the inputs or data 
required by the theory. An essential objective of this thesis is to show 
how a planner might be guided in specifying these inputs. 

The thesis has been written with a view towards its usefulness for 
personnel with a eerie background in probability and statistics. It 
is addressed also to students and practicioners of Operations Analysis 
who may be concerned with the relations of the three statistical methods 
which are considered as tools which may provide quantitative basis for 
executive decision, 

The writer's interest in the possible applications of this theory was 
aroused during the study of Statistical Decision Theory at the U. S. 
Naval Postgraduate School. The need for investigations as to how the 
inputs could be specified was pointed out by LT. R. A. Tucker, USN in 
his thesis: An Introduction to Statistical Decision Functions[1]. 
Tucker's paper presents detailed discussions of the mathematical con- 
cepts involved in the theory and the precise mathematical steps 
required to obtain the solution, It is intended for readers with less 
mathematical background than is required for an understanding of the 
basic work Statistical Decision Functions by Abraham Wald [2]. 
Readers who are interested in the theory and detailed computations 
should refer to Wald and Tucker. 

This paper is divided into five chapters. Chapter I discusses the 
relationship between types of operational testing problems at various 


levels of the Navy. Chapters II and III are examples of applications 
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of statistical methods at the testing agency level. In Chapter IV, an 
example of a guided missile is used as a vehicle of discussion as to how 
the inputs required by Statistical Decision Theory may be specified by 
an office of CNO. A solution to the example is then given. In Chapter 
V the effect of variation of parameters is shown, 

This thesis was written at the U. S Naval Postgraduate School, 
Monterey, California, during the period January-May, 1956. Iam 
indebted to Professor Thomas E. Oberbeck for his continued patience, 
encouragement and most capable guidance while acting as faculty advi- 
sor; and for permission to use solutions he has obtained by program- 
ming the problem for an electronic computer. I wish to thank Professor 
C, A, Magwire for his valuable assistance as second eae and Mrs, 
D, P. Slingerland for her meticulous preparation of this typescript. 
Appreciation is also expressed here to personnel of VX-4 and the U. S. 
Naval Air Missile Test Center, Point Mugu, California, who provided 
much helpful information on the practical aspects of testing. 

The graph on page 8 is reproduced from Burrington and May's: 


Handbook of Probability and Statistics by permission of the publishers, 
Handbook Publishers, Inc., Sandusky, Ohio. 
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GHAPTER I 


INTRODUCTION 


This thesis is concerned with the application of some statistical 
methods to testing programs of the type conducted by OpDevFor or 
similar agencies, operating directly for CNO, Such programs are 
defined as naval operational testing. The thesis does not consider 
applications of these methods to quality control or to develope or 
engineering tests suchas those which might be conducted by the mater- 
ial Bureaus, 

The scope is further limited to those operational testing pro- 
grams in which the object is to obtain an estimate of the probability 
of success of a weapon or weapon system,in future combat. As the 
phrase is used, probability of success can be thought of as generally 
equivalent to hit sro ee percent effectiveness or reliability. The 
term testing program is used to describe any test which consists of a 
series of independent trials conducted to obtain the above estimate. 
The estimate is regarded as providing a quantitative basis for execu- 
tive decision at some level of the Naval organization, 

A fundamental problem in any testing, program is the determination 
of the number of trials to be conducted. For some programs this can 
be difficult’, depending essentially on the magnitude of various "costs", 
It is clear that the number of trials is related to cost. The reconcilia- 
tion of all the costs involved is sometimes difficult, This concept is 


well phrased by Breakwell [3] . Ina sentence taken out of context: 
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A balance is sought between (1) the cost of testing for reliability 
and (2) the risks, because of limiting testing, of either accepting 
an insufficiently reliable product or rejecting a sufficiently reli- 
able one. 
As (1) increases the natural tendency is toward fewer trials; as (2) 
increases the tendency is to desire more trials. Thus, ifa balance is 
to be obtained, the statistical methods used must explicitly relate the 
"cost", the number of trials and the decisions to accept the product or 
weapon, Statistical decision theory provides a rational basis for attack- 
ing this problem but it can only be applied when these costs can be spe- 
cified. Frequently, these costs cannot be estimated by the testing 
agency. Under these circumstances, the costs must be furnished to 
the agency or the agency is compelled to apply a statistical method which 
is not based on such estimates. Chapters II and III are devoted to such 
methods, 

Line officers who are unfamiliar with statistical techniques may 
often be directly concerned with testing programs, Chapter II serves 
to introduce the technique of confidence intervals, which are often used 
in reporting test results, It points out that this technique does not pro- 
vide the planner of the testing program with adequate guidance as to 
how he should specify the number of trials. Chapter III illustrates 
the use of sequential analysis. It is considered applicable to testing 
programs which are conducted to compare improved weapons withan 
existing weapon, 

Testing programs which involve new weapons such as guided mis- 
siles, where the costs in (1) and (2) are high, provide a possible field 


for application of statistical decision theory. It is considered that the 





cost estimates required may be available to naval planners at the CNO 
level. Chapter IV illustrates the planner's role in the application of 
this theory to the problem of testing a guided missile. This chapter 


can be read without reference to preceding chapters, if desired. 





Seer TER tt 


CONFIDENCE LIMITS 


Consider the problem of a fleet testing agency in estimating the 
effectiveness or usefulness of a weapon or weapon system. For exam- 
ple, a destroyer which is equisped witha new or improved yee aioe 
ine weapon, In order to express the effectiveness of this weapon system 
quantitatively, some measure must be used. Suppose the measure cho- 
sen is the percentage of hits achieved by the system ina series of inde- 
pendent trials. This measure is regarded as an approximation of the 
percentage of hits which will be achieved by the system ina future war 
but the actual percentage of hits, or the true value of p as it will be 
called, is an unknown quantity. It is assumed that this true value can 


be estimated by suitable testing. This estimate will be designated by 


a 

Assume that the naval planner has a testing program which simu- 
lates as far as possible the combat conditions under which the system 
might be used. Also, that the number of simulated attacks or trials will 
be fairly large (50 or more) and will be conducted so that they represent 
a random sample cf observations, Eurther assume, that the number of 
trials to be made is fixed by limitations over which the planner has no 
control. 

It is intuitively apparent that the accuracy of the estimate, ee 


will depend upon the number of trials conducted; the larger the number 


oitrials, n , the greater the accuracy of the estimate will be. If we 





designate the result ofa trial by x , then we mayconsider that x can 
have only two values; x = 1 ior success or hit, and x = 0 fora fail- 
mre Oneetiss. ne estimate Pp, mey then simply be the total number of 
hits dividec by the total number of trials. However, this estimate may 
not precisely represent the true p , therefore, a measure of the possi- 
Dle uncertainty in Pe is desirable since the test result will be used as 
a basis for making statements about the true p ofthe system. The 

use of confidence intervals, or limits, provides this measure, This 
technique is best illustrated by an example. 

Suppose 50 trials of the system have been conducted and 15 hits 
scored; thus 2 a = = .30 , What statements can the planner make 
about the true value of p ? By the statistical method known as deter- 
mining confidence intervals, two limits, say L, and 1, , can be 
computed, he can then say that the true value of plies in the interval 
between these limits, but he can make this statement only with some 
erbitrary degree of assurance that itis correct. This degree of assur- 
ance or confidence is expressed by a confidence coefficient a . Its 
value depends upon the degree of confidence the planner desires to have 


- 


when he makes the statement that p lies in the interval Ly to L, ; 


If he wants to be 95% certain then the statement would be 
(A) Probes on L,) = (95 
ee 


where L, = 17 and L 


sion snould be read: "The probability is .95 that the variable limits 


pina 43 , for this example. This expres- 


L, and L, include the true value p between them", This implies 


that there is a 5% chance of being wrong and that the true value of p 





might be outside this interval. A similar statement could be made with, 
say 99% confidence (one chance in 100 of being wrong), but if this degree 
of assurance were demanded the effect would be to spread the limits L, 
aod 86 fartnes apart, thatever(. 15 to 749) . Thus, tae planner would 


2 
memmamnag-e ASSUTeEo about the truta O: ais statement but at the same time 


less certain of the value or p . 

As Mood [4]! points out, (A) should be carefully interpreted because 
it appears that p is a variable when actually itis not, p being a fixed 
value, the true hit probability of the weapon, The variables are L, 


ed #86, With this fact in mind, (A) has the meaning that we are 95% 


2 e 
certain that the interval formed by L, and L, includes p. 





L, and L, are used to represent the following variables: 
= i: Pe 
p(1-p_) 
= ae e 
L, = e. i. lage " 


Ly = Lyle. a 9) 
L, 2 Lo(P ae n) 


These relations show that the limits depend upon the outcome of the 


test, eee the number of trials, n , and the confidence coefficient, 
a . Thus, as pointed out above, if ee joes = 20 and a = er 
then L, and L, are determinedas .i7 and .43 , respectively. 


The planner may feel that the confidence interval .i7 to .43 , 


OC 





computed on the basis of E> 3 from 50 trials, is too large and that 
if more trials had been conducted he could have located p within nar- 
rower limits. If he had obtainea the same outcome of Po =: 3 ontests 
which had consisted of 50, 100 and 1000 trials, respectively, the follow- 
ing Table illustrates the change in the confidence intervalas n and a 


are changed: 


iN®. Of trials Approx. 95% limits Approx. 99% limits 


50 lie ce 15 “= eo 
100 me 20 .19 - .43 
1000 We) <a 33 265 = .335 





Figure | is a chart which illustrates the dependence of the confi- 
dence interval on n and Po For a given value of Po, 74 vertical 
line intersects two curves corresponding toa givenvalue of n . These 
intersections, projected on the vertical axis, are the limits L and 


] 
L,; and the interval formed by these limits spans the true value of p . 
Hence, itis labelled as the p axis. This chart is for the confidence 
coefficient of .95 and clearly shows the effect of increasing n 
Also the number oftrials, n , may be regarded as a function of the 


interval [L., - L,] , a2 and p, . That is,all three quantities 


meot De kKnOwn in Order to determine n . In functional notation. 


n = r(L, ~ L,> a, P 


rae 








Confidence Interval Chart fc 


S, 


Confidence Coefficient a= 


‘oO 


figure ] 


it should be noted that the curves are not very useful for attempt- 
ing to determine, in advance, the number of trials required to give a 
fixed confidence interval of desired length because this woulad mean the 
Outcome of the testing program, Po > would have to be known in 


Zavance,. 


tae 


It is apparent that the use of confidence limits has a place in naval 
testing as a means of stating the results of a testing program ina pre- 
cise manner whichis more meaningful than simply stating the outcome 


oI the program as 2 single number Po But the use of this technique 


does not provide adequate guidance for advance planning to indicate how 





many trials should be run or what degree of confidence should be stipu- 


5 


S OW 


bas 


lated. This technique ed ona fixed number of trials. The planner 


n 


may have chosen the number, in advance, from considerations of time, 


Wy 


ervices SecailFec, etc, , Sab we wism so emphasize that this treory does 


meemaSeveas amy critetic Woon whtcketce blamner catoase sucha chi®tce. 


ct 


ae 
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Swoon ce ANALYSIS 


Sequential Analysis was ceveloved by A. Wald in 1943 for use on 
problems which arose during World War Il. It was widely used in manu- 
facturing establishments for accevtance inspection of '"lots'' of mass 
production items. Toe detailed application of sequential tests to such 
problems is given by the Siatistical Research Group [5]. The principal 
advantage of sequential tests in acceptance inspection is that it reduces 
the amount of inspection required. As shown in [5], the methods of 
secuential analysis can be applied to exneriments. Since certain types 
of fleet testing problems can be thought of as "experiments", the appli- 


gram. will be described, 


i) 


cation of sequential tests toa testing pro 
cuential Analysis can be used when a testing program is to be con- 


° 


or the purpose of comparing the hit probability or the probability 


ry 


ducted 


Oo 


Or success Of @ supposedly improved weapon system with thet of the exist- 
ing system, Testing of this type may be-indicated when it is desired to 
use the test results as a basis for decision to recommend @cceptance or 
rejection of the modified system, for fleet use, 

Seouential analysis does not permit the exact number of triais 
required to be determined in advance, nowever, anaverege or expected 
number may be calculated, Frome naval planning standpoint, ignorance 
of the total number of trials required may pose some problems for sche- 
duling, determination of material requirements, servives and related 


4 


ge, but the use of sequential tests can 


details. This may bea disadventa 
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reswmt -A% possible ecom@my of traeis required to reacha Gecision. This 
economy may vepresent consicerebie savings of time and services. 

su OYGexr to use Sequential Anelysis, the planner must be able to spe- 
City certain quantities or inputs. The following hypothetical example will 
illustvate these inputs and how they might be specified. The example will 
be similar to testing problems of the type faced by fleet testing agencies. 
the tcst resuits, which mignt have been obtained for this hypothetical 
exawo.e are shown in Tabdie 1, 

whAMOLi: A destroyer equipved with a supposedly improved anti- 
submavine weapon is to be tested to determine the hit probability of this 
system (designated as system II). The hit probability of system II is to 
be compared with the known hit probability of otherwise identical des- 


troyers empioying a weapon which has been in service use (designated as 
1 . 


system I). We shallassume the hit probability of system Ito be .2. 


Also, we snall assume thai the cost of the two systems is approximately 


Assume finally, that it is desired to specifya sequentialtest. The 


outcome of testing will indicate whether to accept or reject system Ilas 


eames cure> On the basis of the sample ot trial runs, Therefore, the 
input. of the Sequential Test must de carefully specified. YTnese inputs 


unicuc.iy define the sequentia: test: 


(a) a The hit prcoability which would make the new system II 


ro 


"Unacceptao.e"’. 
(>) P> - The hit probability which would make the new system Il 


"Acceptabie' 





(cj) & - The maximum ciloweble risk or probability of rejecting a 
MeW Ss Yoteeuwe. .. it 2 cit m-obability Po OF better, 
(c) jp - The maximum allowable siek or probability of accepting 


system II if it hes a hit probability of P, Gz Jess; 








Ce ee or. LION OL  §5.,4Pp2,X ,f9y In this exarmle, the first 
inDut, >, , is probably the easiest to select, Since 5p, isthe 
+ + 


hit probe dility of 2 new system which would make it uneccevdiable, it 

mat any new system would not be desirable it it were no 
better than the Beas Ome, ieee ls, om its nit aaa was no better 
than that of system I, which we have 2ssumed to be about -2. Hexce 


fee BS! = 3c. 


Tne second input, might be selected by reasoning 2s follows: 
Py 2 o o 


At fixsst thought it would appezr that any system with 2 hit probability 


a 


oY 
: 


rea than that of existing systems is "acceptable''; however, since 


cm 
ww 
the test will consume time and money it would not be logical to accept 


a system with « hit probability only slicvhily greater than existing sys- 
tems; say, 29 between .2to .3 . Onthe other hand it might be 
arcued that the new system should be at least twice as good as the old 
to justify expense of conversion, thetis, anincrease oz: p to “4 


i L) 


hte 


woulca justify the expense of tests and installation of the new system if 
ane insuts CA anc = represent the probabilities of making a 
wWwrony Gecision and these risks are unavoidabit, The values of & 


and /# are smalland not necessarily ccual 


C~ is the probability of rejecting the new system if is has a hit 


4 





probability or .4 . Since it would be undesirable to reject a new system 
which is, on the cverage, twice as good 2c the existing one, then the pro- 
‘bability of making such an efrcr shou.d be mede very small. Tf2 risk 

of one chance in 100 can be tolerated then G& would be .O1 
, On the other henc, meth Srobability of accepeitg < new system 


1.4¢ Jee ee Hit Drobabilir p. . Since it is possible for the test to lead 


Q, 


us to SGuca a wrong decision it would mean that we were accepting a new 


os 


a 


Sycte.. WHECI Was agtually no bette>, vernaps worse, in terms of EI 


’ > os xe 


roba@tinty than the existing cre, =tence, we want to make the probabi- 


Oo 


lity cl making suchan error :mmallalso. But, we can toletate a greater 


cisx of this error tran we can of rejecting a better system, so pP can 
be m@ce iarger than A . In other words, accepting system I] when 

it has the same hit probability as the old system is not too serious from 
amilitary standpoint. Thereiore if fe were selectedas .10 , this 
Wou.d be taking one chance in ten of making suché Pete CC. S107) peo 


« 5 ae 5 4 
iliustretive purposes, choose Q = ,01 and = ae) 


S@ece either error is sossitdle, it would seem desirable to have the 


~ 


paees, CKO and f/f , oO: meine such errors as small as possible, 
Mhetis, make CA and /f© even smaller than the values chosen above. 


It will be seen that demanding smaller risks will result in having to make 
Seesester number of tridis ana this number can become unacceptably 
laroe; therefore some risk must be -Slerated to avoid 2 prohivitive num- 


- we 


per ct (rials. 
raving selected the inputs 


od 


Pp; ees Po = .4 A = 208 fe 


oS 





1t is mathematically possible to determine what is definedas a Power 
Curve, wa (p) . This curve will represent the probability of rejecting 


a 


system llas afunctionof p . It will have the following general shape: 





Sibere 2 
ane Ordinate at any point p = p' , represents the probability, 
« (p') , of rejecting the new system when its hit probability is p' . 
ft p=3b,2.4 , wip) has the value of A&A = .01 , .which is the 


) 


roo dility of xejecting system I] when its hit probability is .4 . At 
this point we are taking one chance in 100 of rejecting system II when its 
myue p is equalto .4 . Wote thatif system IJ] hasa p> .4, we 
nave a still smalier probability of rejecting it. 

At the point p = P) =- 2 we nave avery high probability of reject- 


ing the new system and since (one minus the probability of rejection) is 


equal to the probability of acceptance, then [1 - f (p,) | = SS 





This is the risk we are willing to take in accepting the new system when 


it is no better than the existing one, 


Betweer mp. fei.) the new system will betrejected with probi-= 
e . . a) ene ° e 
PitiesS Vaanyeno from 1 - P" to - the probability of rejection 


decreasen# =s we approach tas "acceptable" hit probability of 1.4 . 
Tre Power Curve need not actually be produced in order to make 
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The Weight Function can now be graphed as; 


W(p, ep) 
j 
Value 
units 
(oe) SI IT NTI TS ELE — a Yo 
O o. 66 J.0 p 


(a) Case where Decision 
to Convert is made 


w/(p, o,*) 
1 
O 6 (1-2) hia Ie 


(b) Case where Decision 
to Develop is made 


Linear Weight Function 
Showing the Parameter 2 


Figure 


The value of 9 


example, 92 is chosenas ,.33 . 


is at the discretion of the planner. 


15 


For this 


Another possible form of Weight Function 


It should be noted that the linear form of the Weight Function 


resulted from the way in which it was assumed that the cost of wrong 


decisions could be assigned. 


As Wald [2] points out, the Weight Func- 


tion is the most difficult input of the Statistical Decision Problem to deter- 


mine, 


many practical problems. 


He further points out that a Simple Weight Function is suitable for 


A simple Weight Function has only two values, zero or one, 
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w(p. o,") wip, o,') 


1 . 1 i 
I 
Value | | 
units l 
I 
O ee ee ae Say ay 0 
O o 6 2.0 Pp O @ (1-2) vee, - 
(a) Case where Decision (b) Case where Decision 
to Convert is made to Develop is made 


Simple Weight Function 


Figure 16 


: and d._° are exactly the sameas 


The meaning of 6, p, 0, d, 2 


before. 

The difference between this Simple form and the Linear form is 
that the cost of wrong decision is not proportional to the difference 
between p and 8 . There is no penalty to be assessed fora missile 
whose true probability of success lies between 9 and (1-90) . But, 
when the decision to convert is made and p< 0d, the penalty is the 
entire cost of conversion, Similarly, when the decision to develop is 
made and p > (1-98) the penalty is the entire cost of further devei- 
Opment, 

The Simple Weight Function has been illustrated so that compari- 
sons of solutions using Simple and Linear Weight Functions can be mace, 
Input V Cost of testing of Cost Function. (C) 


This input is not as difficult to specify. In many testing programs 


ad 





the cost of each trial in the program is the same. When this is the case, 
then the cost of testing is proportional to the number of trials. 

As inthe other costs, the naval planner must be able to estimate 
this cost of testing. This would include such items as the cost of the 
missile itself; contractors services required for the tests, etc. Cris 
therefore determined by a cost analysis of charges directly applicable 
to the testing program fora specific missile. It is not considered that 
the cost of operating the testing agency for the duration of the testing 
program is a cost which would be included here, 

Suppose the cost of each trial for the missile in this example is on 
the order of $50,000. As it is used in Statistical Decision Theory, C 
is expressed in the same value units as the Weight Function. For the 
Weight Function with the maximum value of $10! = | value unit, then 
define c = C/W = .005 value units. 

The effect of varying the ratio C/W will be discussed in the next 


chapter. 


(B) Solution of The Example 

The preceding discussion has indicated how the inputs required for 
a Bayes Solution of a Statistical Decision Problem may be specified. 

The solutions are given inthe form of Table 2, using the Linear 
Weight Function and Table 3, using the Simple Weight Function. 

By way of summary of the problem to which the Tables are solu- 
tions, we shall briefly review the special circumstances required of the 
testing program and the four inputs to the problem. 


(a) Recall from Page 26 that a suitable testing program will co--~ 
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(b) 


(c) 


(d) 


(e) 


sist of independent trials, the outcome of whi 


cess or failure. The estimate P. is the nu: 


divided by the total number of trials, 


Input I. The specification that the true p 


take on any value in the interval [0, 1] ané 


p are equally likely. 


Input II. The final or terminal decisions anc 


to the preassigned value of 89 = .5 . 


t 
cy 


d," - the Decision to Develop 


InpuG tie 


- the Decision to Convert 


The cost of wrong decision or We 


both Linear and Simple forms. 


Lue ke 
W(p, a - Cost or penalty when dy is a 


W(p, am) - Cost or penalty when any is a 


The maximum value of W is alsisumed to be = 


~ 


ialjewle)) Zinekel) — e) ay 5). 


rapt Ive tnerecost ot testing on Gosu une f1 
Testing was assumed to cost $50, 000 per tria 
c = $50,000/$10' = .005 
Note that c 
the same, i.e., if we had assumed W to be $ 
siete, | ie GE TOM) velguena, 


is again .005 


has the same value as long as t. 


is either suc= 


ey of successes 


ne missile can 


at all values of 


eit re lacvion 


function (W); 


ao decision 
ng decision 


(1 value 


\C) 


Tice, 


atio C/W is 


and Cost per 


Explanation of Table 2 (Linear W); page 40 


Any cell in the table can be identified by a pair of: bers or coor- 


dinates, say (i, j) , where 1 denotes the number of ituresand j 


hy) 





er 


mJy O 


Otdw Grrr y 


NUMBER OF — 


.035704 


,OLS66 |) 00386 


.00247 | .00960 |,. 
00247 | .00950 
00069 | .00414 |, 
200069 | .00314 


Solution Using Linear Weight Function 


Table 


40 
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Pree 
0024-7 


a cee ad | 
02254 | 
01926! 


00069 
00059 


003840 


003/40 








the number of successes which have been observed in {i + j) trials. 
The numbers i and j are the numbers designating each row and col- 
umn respectively. Thus, row two and column three locate a specific 
cell after trials which yield two failures and three successes, 

Every cell contains an upper and lower entry. Each entry repre- 
senfs a cost expressed in value units, If desired, these entries may be 
converted to dollars by multiplying by the dollar equivalent of a value 
unit ($10! in this example) . The upper entry is the expected cost (in 
value units) if no further trials are made and a Terminal Decision to 
Convert or Develop is made onthe basis of trials thus far, The lower 
entry is the expected cost (in value units) if trials are continued and a 
Terminal Decision to Convert or Develop is based upon the result of 
further trials. 

The relation between the upper and lower entries means that, in 
any cell where the lower entry is smaller than the upper, the expected 
costis less to continue testing than it is to make a decision at this point, 
Notice that the cells in which the lower entry is smaller than the upper 
are enclosed by the dotted line. 

It is this dotted boundary which enables the planner to determine, 
in advance of testing, the limits on the number of trials required, The 
Table actually yields the minimum as wellas the maximum numbe:z of 
trials that might be required. These maximum limits are of interest 
to the planner. For our example the minimum number of trials, Nn ; 
is 2 andthe maximum number, N eco: 


M 


We shall now illustrate the use of the Table during testing. Before 


4) 





starting the testing program, there are, of course, no failures and no 
successes and we start inthe (0, 0) position. The lower entry is 
smaller than the upper, therefore a trial is conducted. If this trial is 
successful, move tothe (0, 1) position (0 failures, 1 success); if the 
trial is a failure move tothe (1, 0) position (1 failure, 0 successes). 
Hither outcome, however, leads to a position within the dotted line so 

a second trial is conducted. The process continues until we are led to 

a position outside the dotted line. To get toa position outside the dotted 
line will require at least 2 trials and at most 9 trials. Hence, Ne = oe 
and Nyy = 9 . 

To confirm these numbers note that either 2 successive failures 
or successes will lead outside the line. Or, alternate success and 
failure will lead one diagonally dgewn the Table to the (4, 4) position. 
This requires 8 trials. Once here, either another success or failure 
will lead one to a position which is outside the dotted line and not shown 
onthe Table. Hence, the maximum number of trials is 9 , and is the 
sum of the largest row and column designator plus one. 

The testing program is thus completed when experimental results 
have led to crossing the dotted line of the Table as described above. 
Then the value of Pe is the number of successes divided by the number 
of trials. A comparison of De with the preassigned 9 leads to the 


& 


Terminal Decision ae (convert) if Pw 89; and to the Terminal 


Decision a (develop), if Pe < 0 See Pe = 6 . eithenm deciciom 


mam be made, 


Table 3 (Simple W) , is used in the same manner as Table 2 


cae 


Note that the Nu is greater for the Simple W , specifically Nu = 
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CHAPTER V 


VARIATION OF INPUT PARAMETERS 


Tables similar to Table 2 and Table 3 may be prepared for other 
values of the basic parameters c and @ - for prescribed Weight 
Functions, From these tables Nyy may be determined. Recall 
from page 38 that it is only the ratio C/W , which we called c , 
which is a parameter of the Tables. 

The curves in Figures 17 and 18 show a plot of Nui for sets of 
values of the parameters as indicated. Figure 17 is for the Linear W 
and Figure 18 for the Simple W . These curves are plotted on semi- 
log paper. This was done since solutions are more easily obtained 
ae c is varied by a factor of 10 thus the semi-log plot provides a 
more extended graph. The tables, from which the values of Ny were 
found, were obtained by means of a CRC model 102-A electronic digital 
computer, atthe U. S Naval Postgraduate Scheel Such Tables may 
also be calculated with the aid of the mathematical tables in Pearson [6]. 
For details of such calculations see Tucker [1]. 

Note, from Figure 17 for any given value of 0 ,as c decreases, 
the number of trials increases. This shows that if the cost of testing 
is decreased more trials are made before a decisionis reached. Con- 
versely, as the cost of each trial approaches the maximum value of W 
(maximum cost of wrong decision}, fewer trials are conducted. Or, 


briefly, the more costly the testing, the fewer the number of trials. 


Notice, further, that the number of trials required is very sensitive 
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to changes in 0. , particularly when 0 isclosetothe value of 6 . 
As 90 approaches @ , the number of trials increases, A value of 20 
close to @ requires the theory to be more discriminating and this, in 
turn, requires more trials. 

Figure 18 is similar to Figure 17. The only difference being, 
that for given values of 0 and c in Figure 17, the corresponding 
point in Figure 18 givesa larger value of n . In other words, for 
the same 0 and c_ , the Simple W always results in more trials 


being required than indicated by the linear W, 


Summary 

Statistical Decision Theory has been applied toa special problem, 
and two different ways in which the Weight Functions could be specified 
have been suggested. A value of Nuy ; the Maximum) number 2 eaame 
required, is indicated by the solution. This number can be useful to 
Naval planners as a basis for decision as to the number of missiles to 
be supplied to the testing agency for the tests. It should be noted that 
the value of Nyy given by the solution will have to be increased to pro- 
vide for those trials in which missiles are expended but, for some rea- 
son, the trial in invalidated. 

It is assumed that naval planners contemplating the use of Decision 
Thedry would have assistance from statisticians or mathematicians, 
Solutions can be extremely time consuming to produce manually, hence 
electronic computers will increase the practical usefulness of the theory. 


Dr. T. E, Oberbeck has obtained solutions for special cases using sim- 
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ple and linear Weight Functions, by programming the problem for the 
CRC model 102-A computer. 

For the use of naval planners, a more complete set of curves than 
those given in the chapter would facilitate the application of the theory. 
Such a project awaits further effort by research workers in this field. 

In cases where the planner is unwilling to specify an a priori prob- 
ability density function associated with p (see ee 20 | » a Minimax 
solution of the problem may be obtained. For results on the average 


maximum number of trials see Breakwell [3]. 


re be reported at the Fourth Annual Meeting of the Operations Research 
Society of America, in May 1956, 
48 
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APPENDIX 


The graphs for sequential tests shown by Figures 6, 7 and 8 of 


Chapter III are constructed as follows. 


p Pee i dy |  peane ar crm 
ae a In _ ] ] a lee2 l 


is calculated. Where n number of trials 


number of successes (hits) 
(1 - p) 


6) 
Howe 


a 


(2) The criteria defining the three regions are: 


(a) If » R < a ae ’ accept system II 
(b) If » R > +58. ; reject system II 
(c) If vx < » R < jsf , continue testing 
mor Case A : io ai au Ln 4 A = ,01 
qj = 8 G5 = 0 ‘e = l 


then the inequality (a) is 


1S AG: | alee a ee 
8 3 = oa 1-H oa 


Solving for s_, by logarithms gives 


Si ee ete ee on 
Values of s and n_ which satisfy this inequality, define the 
acceptance region, 


Similarly, inequality (b) is 
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Ss 1). 


3 4 | _ 
35 > ma ae hs. ’ eR RSE a) ae? 9 0 


“8 
Solving for s_ gives 


<= -4,58 + :293n 


3) 


Values of s and n which satisfy this inequality, define the 


rejection region, 


(3) The parallel lines in Figure 6 are obtained by plotting the straight 


lines 


S = Cao) ot Eo om 


-4,58 + eon 


6) 
tt 


In the same manner, the straight lines in Figure 7 and 8 are obtained. 


For Case Te. Ss = A05 + ~293n 
Ss = ete + .293n 
fmoricase C : S = 1,28 + N Sis ig) 
Ss = des | + nSisi {sal 
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